Boundless projected interactive virtual desktop

ABSTRACT

A method for creating a boundless projected interactive virtual desktop, wherein the interactive virtual desktop comprises an adjustable image of a projected portion of an area associated with at least one desktop of a computing device is provided. The method may include integrating a projector and a motion sensor into a device. The method may also include capturing at least one of a location, a change in location, a change in direction, or a change in orientation associated with the device. The method may include computing a projected image. The method may also include coordinating the computed projected image across at least one application running in the device. The method may further include projecting a view of a portion of an area associated with the coordinated projected image, wherein the projected view comprises an interactive virtual desktop. The method may additionally include adjusting the projected view based on a criteria.

BACKGROUND

The present invention relates generally to the field of computers, andmore particularly to projectors.

A projector or image projector is an optical device that projects animage (or moving images) onto a surface, such as a projection screen.Currently, projectors may be available as a handheld projector (alsoknown as a pocket projector, mobile projector, pico projector or minibeamer). A handheld projector may even be embedded in smartphones. Ahandheld projector is technology that applies the use of an imageprojector in a handheld device, such as mobile phones, personal digitalassistants, and digital cameras.

SUMMARY

According to one embodiment, a method for creating a boundless projectedinteractive virtual desktop, wherein the boundless projected interactivevirtual desktop comprises an adjustable image of a projected portion ofan area associated with at least one desktop of a computing device isprovided. The method may include integrating a projector and a motionsensor into a device. The method may also include capturing at least oneof a location, a change in location, a change in direction, or a changein orientation associated with the device from the integrated motionsensor. The method may include computing a projected image based on thecaptured location, the captured change in direction, or the capturedchange in orientation of the device relative to a projected surface. Themethod may further include coordinating the computed projected imageacross at least one application running in the device based on thecaptured location and an orientation of the projector, wherein theprojected image is associated with the projection surface. The methodmay also include projecting a view of a portion of an area associatedwith the coordinated projected image, wherein the projected viewcomprises an interactive virtual desktop including a plurality ofelements comprising application images. The method may additionallyinclude adjusting the projected view of the portion of the area based ona criteria.

According to another embodiment, a computer system for creating aboundless projected interactive virtual desktop, wherein the boundlessprojected interactive virtual desktop comprises an adjustable image of aprojected portion of an area associated with at least one desktop of acomputing device is provided. The computer system may include one ormore processors, one or more computer-readable memories, one or morecomputer-readable tangible storage devices, and program instructionsstored on at least one of the one or more storage devices for executionby at least one of the one or more processors via at least one of theone or more memories, whereby the computer system is capable ofperforming a method. The method may include integrating a projector anda motion sensor into a device. The method may also include capturing atleast one of a location, a change in location, a change in direction, ora change in orientation associated with the device from the integratedmotion sensor. The method may include computing a projected image basedon the captured location, the captured change in direction, or thecaptured change in orientation of the device relative to a projectedsurface. The method may further include coordinating the computedprojected image across at least one application running in the devicebased on the captured location and an orientation of the projector,wherein the projected image is associated with the projection surface.The method may also include projecting a view of a portion of an areaassociated with the coordinated projected image, wherein the projectedview comprises an interactive virtual desktop including a plurality ofelements comprising application images. The method may additionallyinclude adjusting the projected view of the portion of the area based ona criteria.

According to yet another embodiment, a computer program product forcreating a boundless projected interactive virtual desktop, wherein theboundless projected interactive virtual desktop comprises an adjustableimage of a projected portion of an area associated with at least onedesktop of a computing device is provided. The computer program productmay include one or more computer-readable storage devices and programinstructions stored on at least one of the one or more tangible storagedevices, the program instructions executable by a processor.

The computer program product may also include program instructions tointegrate a projector and a motion sensor into a device. The computerprogram product may also include program instructions to capture atleast one of a location, a change in location, a change in direction, ora change in orientation associated with the device from the integratedmotion sensor. The computer program product may include programinstructions to compute a projected image based on the capturedlocation, the captured change in direction, or the captured change inorientation of the device relative to a projected surface. The computerprogram product may further include program instructions to coordinatethe computed projected image across at least one application running inthe device based on the captured location and an orientation of theprojector, wherein the projected image is associated with the projectionsurface. The computer program product may also include programinstructions to project a view of a portion of an area associated withthe coordinated projected image, wherein the projected view comprises aninteractive virtual desktop including a plurality of elements comprisingapplication images. The computer program product may additionallyinclude program instructions to adjust the projected view of the portionof the area based on a criteria.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

These and other objects, features and advantages of the presentinvention will become apparent from the following detailed descriptionof illustrative embodiments thereof, which is to be read in connectionwith the accompanying drawings. The various features of the drawings arenot to scale as the illustrations are for clarity in facilitating oneskilled in the art in understanding the invention in conjunction withthe detailed description. In the drawings:

FIG. 1 illustrates a networked computer environment according to oneembodiment;

FIG. 2 is an operational flowchart illustrating the steps carried out bya program for projecting a boundless interactive virtual desktopaccording to at least one embodiment;

FIG. 3 is an exemplary illustration of the projected desktop imageaccording to at least one embodiment;

FIG. 4 is an exemplary illustration of movement of the device adjustingthe projected desktop image according to at least one embodiment;

FIG. 5 is an exemplary illustration of coordination of multiple devicesinteracting with different subsections of the projected desktop imageaccording to at least one embodiment; and

FIG. 6 is a block diagram of internal and external components ofcomputers and servers depicted in FIG. 1 according to at least oneembodiment.

DETAILED DESCRIPTION

Detailed embodiments of the claimed structures and methods are disclosedherein; however, it can be understood that the disclosed embodiments aremerely illustrative of the claimed structures and methods that may beembodied in various forms. This invention may, however, be embodied inmany different forms and should not be construed as limited to theexemplary embodiments set forth herein. Rather, these exemplaryembodiments are provided so that this disclosure will be thorough andcomplete and will fully convey the scope of this invention to thoseskilled in the art. In the description, details of well-known featuresand techniques may be omitted to avoid unnecessarily obscuring thepresented embodiments.

Embodiments of the present invention relate generally to the field ofcomputers, and more particularly to projectors. The following describedexemplary embodiments provide a system, method and program product to,among other things, provide a boundless projected interactive virtualdesktop. Additionally, the present embodiment has the capacity toimprove the technical field of image processing by enabling theprojection of desktop images of an unbounded surface of a desktop onto asurface. Furthermore, the present embodiment has the potential toconserve resources. As such, battery power will be conserved by thecombined adjustment of total lighting and lens adjustment, resulting ina preservation of the luminance at the cost of the projection size.

As previously described, projectors may now be available as handheld orpico projectors and may even be embedded in smartphones which havesufficient storage capacity to handle presentation materials but littlespace to accommodate an attached display screen. Handheld projectorsinvolve miniaturized hardware and software that can project digitalimages onto any nearby viewing surface. However, such handheldprojectors may not enable a wide user interface, but rather may justprovide a mirroring of the screen into a projected surface.Additionally, the current handheld projectors may suffer from highenergy usage and therefore, may require frequent recharging of thebattery.

Furthermore, a common problem exists with respect to the use of aprojector or a standard screen. Users of such devices may struggle withmulti-tasking, copying or pasting functions or other interactions whichmay require being coordinated across applications. For example, complexorganization, such as filing a large number of documents may bedifficult and may require a user to navigate among many dialogs andchoice lists on a small screen. As such, it may be advantageous, amongother things, to implement a method of projecting an image that may bepanned, scaled, and rotated according to the location and orientation ofthe device's projector.

According to at least one implementation, the present embodiment mayinclude a projector device that includes a motion sensor, and softwarethat may use the motion sensor in the device to pan, scale, and rotatethe projected image as a boundless interactive virtual desktop (i.e., anunbounded interactive virtual desktop) according to the location andorientation of the device's projector. As such, the unboundedinteractive virtual desktop is workspace, at least a portion of which,exists only in the memory of the device but of which a bounded area or aportion of a bounded area may be visualized through the projector and inwhich elements may be moved infinitely in any direction. Morespecifically, a virtual unbounded desktop is a desktop that has thecapacity to have at least a portion of itself exist in memory alone,i.e. at least a portion of it exists in memory and may or may not bedisplayed or projected on or via a tangible medium. For example,according to at least one implementation of the present embodiment, theposition of each element is stored relative to a center position for thedesktop, and a representation of a portion of a desktop projected andportrayed as an bounded image or portion of the unbounded area, onto asurface, such as a wall or a desktop. Furthermore, the projected portionof the desktop can be changed based on the detection of a movementassociated with the projector device. Therefore, a detected change inmovement associated with the projector device corresponds to the surfaceof the projected portion of the interactive desktop by revealing apreviously unrevealed portion of the desktop associated with theprojector device. For example, movement of the projected device to theright may reveal a previously projected or unrevealed image associatedwith the desktop. In one embodiment, the movement to the right, maymimic a “swipe” feature linked to a desktop of a computer screen ormobile device screen. Furthermore, according to various embodiments ofthe present specification, the projected portion of the desktop may be aprojected portion of the desktop associated with the projector device orit may be a projected portion of a desktop associated with a computingdevice connected to the projector device.

Additionally, two or more projector devices may be used in conjunctionwith one another to portray a contiguous, larger projected portion ofthe virtual interactive desktop. For example, the portrayed projectedportion of the combined desktops of the projector devices may allow fora larger area of each of the desktops to be portrayed rather than wheneach projector device is operating in isolation.

Furthermore, a user may interact with the virtual desktop by usinggestures, the on screen display, and buttons on the device. For example,as a user moves the projector device containing the motion sensor, theportion of the boundless desktop being projected may be adjusted. Forexample, movement to the right may portray a different portion of theprojected desktop or such movement may include the portrayed portion ofthe desktop before the movement in addition to portraying the newportion of the projected desktop. Furthermore, the desktop may becapable of displaying a number of “elements”, such as documents, images,folders, video, web browser tabs, application windows or applications.

According to at least one implementation, the complete desktop may beviewed on a screen or any large viewing surface, such as a wall.Additionally, by holding the device far from the projection surface, thearea illuminated by the projector may be increased, however, thestrength of the projection may be decreased, possibly to the point thatit is not visible. As such, according to at least one implementation,boundless virtual surface may be created using only a small andrelatively low power projector which may be embedded in the device, andmay adjust the strength of the projection based on available batterylife without sacrificing the effective strength. For example, the usermay move the device closer to the surface and, although the illuminatedarea is then smaller, it may be just as bright with less powerconsumption in such a situation. Additionally, according to at least oneimplementation, multiple users may be allowed to interact with theelements simultaneously. Also, the relative size of the projection areamay differ depending on the height of the device; however, the locationof elements may be preserved relative to the projection surface, not thedevice.

Various embodiments of the present specification may maintain theprojected image fidelity as well as preserve the location of projectedelements relative to the projection surface. Additionally, a low powerprojector may be utilized to illuminate a subsection of a largeraugmented reality while providing real-time adjustment of the projectedimage; real-time adjustment of the projected image based on the motionsensor; coordination of multiple devices interacting with differentsubsections of a projected virtual desktop; and conservation of batterypower by combined adjustment of total lighting and lens adjustment,resulting in a preservation of the luminance at the cost of theprojection size. Furthermore, gestures made with the projection deviceitself may preserve relative positioning on the projected surface whileallowing selection, drag and drop and other operations.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The following described exemplary embodiments provide a system, methodand program product to provide a boundless projected interactive virtualdesktop. Embodiments of the present invention may calculate theprojection to take into account a number of variables, all eitherabsolute or relative to a marker. For example, the location of thedevice may be calculated in three dimensions. As such, the viewing areamay be panned and the projection zoomed. Also, the orientation of thedevice may be calculated in three dimensions so that the image of theviewing area may be stretched and distorted as if the device was flat.Furthermore, the placement of the objects on the surface may becalculated. As such, an object placed on the surface may be detected bya camera or a camera array, either built into or supplemental to themain device. Then, the image may be corrected so that the image may notbe magnified by being closer to the device when the object is placedbetween the surface and the device. Additionally, to further improve theaccuracy, the device may be used in conjunction with table mountedcameras or other sensors (such as an array of infrared light (IR)devices) capable of pinpointing the location and position of the devicein three dimension.

Implementations of the present embodiment may also allow for userinteraction. For example, a series of gestures may be made with thedevice itself to select and interact with elements on the tabletop. Someof the main interactions involve picking up and dropping items, andusing the device screen and gestures to interact with the device. Inparticular, a user may flick the device forward to put an item down(which may create a stack); flick it up to pick up an item from thetabletop; or flick it to the side to shuffle through a stack. As thedevice is moved, the movement may be tracked through motion sensors inthe device. Such motion sensors may detect surfaces nearby or suchmotion sensors may interact with devices on or near the surface.According to at least one implementation, the projected image may berecalculated to represent the area of the image which corresponds tothat part of the surface, and potentially skewed and stretched usingexisting technology to account for the angle of the device relative tothe surface(s). Additionally, a user may also pick up a single multiplepage element, such as a book or document and then use a flick to theside to move through the pages of the document.

Various embodiments of the present specification may also allow forobject pinning. As such, virtual objects may be pinned to physicalobjects to create hybrid stacks. For example, a virtual element placed“on” a coaster on the real tabletop may move when the coaster is moved.An element, such as a document, photo or video that is placed on a realsheet of paper may move with the real paper as if it is in a stack ofpaper. As such, the stacks may be moved when in view or out of view ofthe camera. According to at least one implementation, if the physicalitems are out of view of the camera, the device may first recognize andrecord their new locations when the physical objects are first observedin a new location. However, if the old location is viewed by the camerabefore the new location is located, the stack may be placed in a virtual“bin” temporarily until the location can be reassigned. Then, the usersmay sort through the bins and stacks on their device independent oftheir physical locations in a stack and bin view and may use this viewto further organize documents even once they have left the location ofthe tabletop. As such, a tabletop may be saved and recreated in a newlocation. To facilitate this, the user may take a bin (i.e., any stackwhose physical pin or counterpart is not yet found) and place it on thetable in a new location.

According to at least one implementation, the present embodiment mayallow for on screen interaction. Therefore, when the device is pointedat an element on the table but no element is “in hand” or picked up yet,the element being pointed to or a representation of that element, suchas a specific icon may be displayed on the device. Then the user mayinteract with the element on screen. For example, the user may interactwith the element by editing it, reading it, or navigating within thatelement. When a user picks up an element, the element is “locked” to thescreen until the user puts the element down or uses other on screennavigation to move away from it. Additionally, a display on the devicescreen may allow for further interaction with elements and stacks bylisting elements, stacks, and elements not in a stack and allowingoperations on the elements and stacks. Furthermore, the on screendisplay may appear when the device is not pointed at an element or whena button is pressed on the device or a special gesture made. The presentembodiment may allow a stack to be converted to or added to a folder,collection or set in an online document or content management system orfile system using gestures or the on screen display. For example, anentire table may be converted to a library or set of folders with asingle action, mirroring the tabletop organization in a contentmanagement or file system. A stack may be added to a folder so furtheritems in the stack will be added to the folder automatically and a stackmay be linked to a folder so items in the folder may automatically getadded to the stack. All actions on the on screen display may be takenwith gestures and the device may allow for customization, such asassigning gestures to actions and programming of new gestures.

According to the present embodiment, a device itself may be handheld,wearable or implantable and can include a projector, motion sensor, onscreen display and specialized software. These may be physicallyseparated into different modules working together in variousimplementations. Furthermore, the present embodiment may allow forluminance adjustment. As such, the projector may also include lenseswhich may be used not only to focus the projection, but to downsize orupside the area of projection while preserving luminance. Using thismechanism, the illuminated area may be reduced by reducing the overalllighting strength from the device without reducing luminance. Thisluminance adjustment may be used to preserve battery life or to keep theluminance constant as the device is moved closer or farther from thedevice.

Additionally, multiple users who are each using the present embodiment,may all simultaneously interact with the elements on the virtualtabletop. When an element is modified the modifications may becommunicated through near field communication, through a server or cloudservice or through network communication between the devices. As such,multiple devices may interact either with radio antenna or otherpositioning sensors to improve their position and orientation relativeto each other. As the projection areas become close and overlap, camerason the devices may be used to improve the accuracy of device locationand orientation detection by monitoring the image projected by the otherdevices relative to the representation of the tabletop in memory.Additionally, visual tokens or elements outside of the human visualspectrum may be embedded in the projection to assist multiple devicescoordinating their projections.

Referring now to FIG. 1, an exemplary networked computer environment 100in accordance with one embodiment is depicted. The computer environment100 may include a computer 102A, 102B with a processor 104A and a datastorage device 106A that is enabled to run a Boundless ProjectedInteractive Virtual Desktop program 108A. According to at least oneimplementation of the present embodiment, the computer 102A, 102B mayalso include a motion sensor 116 and a projector 112. The motion sensor116 may include any mechanism to capture the relative motion of thedevice. Additionally, according to at least one implementation of thepresent embodiment, the motion sensor 116 may be capable of detectingthe position of the device 102A, 102B at one point in time, and then theposition of the device 102A, 102B at a later point in time. Furthermore,according to an alternate implementation, in addition to a motion sensor116, which may aide with movement of the device relative to the surface,the device 102A, 102B may use techniques from using either an active ora passive autofocus detection system (such as contrast or phasedetection) to detect the relative angle of the plane of the projectedsurface relative to the device. As such, multiple points can be sampledfor optimal focus, deriving the distance to multiple points in the planeof projection and therefore may allow for the computation of thedistance to multiple points and recreation of the plane's geometric inthe device's memory.

The networked computer environment 100 may also include a communicationnetwork 110 and a server 114 with a processor 104B and a data storagedevice 106B that is enabled to run a Boundless Projected InteractiveVirtual Desktop program 108B. The networked computer environment 100 mayinclude a plurality of computers 102A, 102B and servers 114, only one ofwhich is shown. The communication network may include various types ofcommunication networks, such as a wide area network (WAN), local areanetwork (LAN), a telecommunication network, a wireless network, a publicswitched network and/or a satellite network. It should be appreciatedthat FIG. 1 provides only an illustration of one implementation and doesnot imply any limitations with regard to the environments in whichdifferent embodiments may be implemented.

As will be discussed with reference to FIG. 6, client computer 102A,102B and server 114 may include internal components 800 and externalcomponents 900, respectively. Client computer 102A, 102B may be, forexample, a mobile device, a telephone, a personal digital assistant, anetbook, a laptop computer, a tablet computer, a desktop computer, aserver computer, or any type of computing device capable of running aprogram, such as a Boundless Projected Interactive Virtual Desktopprogram 108A.

A program, such as the Boundless Projected Interactive Virtual Desktopprogram 108A, 108B may run on the client computer 102A, 102B or servercomputer 114. As previously described, the boundless projectedinteractive system may include a projector 112 and a motion sensor 116integrated into a mobile device, such as client computer 102A, 102B thatincludes software, such as the Boundless Projected Interactive VirtualDesktop program 108A. The Boundless Projected Interactive VirtualDesktop program 108A, 108B may use the motion sensor 116 in the device102A, 102B to pan, scale, and rotate the projected image as a boundlessvirtual desktop according to the location and orientation of thedevice's projector. Additionally, a user may interact with the projectedvirtual desktop by using gestures, the on screen display, and buttons onthe device 102A, 102B. For example, as a user may move the projectordevice 102A, 102B containing the motion sensor 116, the portion of theboundless desktop being projected may be adjusted. Furthermore, thedesktop may be capable of displaying a number of “elements”, such asdocuments, images, folders, video, web browser tabs, application windowsor applications. The Boundless Projected Interactive Virtual Desktopmethod is explained in further detail below with respect to FIG. 2.

Referring now to FIG. 2, an operational flowchart 200 illustrating thesteps carried out by a program for projecting a boundless interactivevirtual desktop according to at least one embodiment is depicted. At202, a projector and a motion sensor are integrated into a mobiledevice. The Boundless Projected Interactive Virtual Desktop program,108A, 108B (FIG. 1) may be implemented as running on a client computer102A, 102B (FIG. 1). As previously described, the Boundless ProjectedInteractive Virtual Desktop system may include a projector 112 (FIG. 1)and a motion sensor 116 (FIG. 1), integrated in a device, such as, butnot limited to an End User mobile device 102B (FIG. 1), a computer 102A(FIG. 1) a smartphone, or a tablet that includes a motion sensor 116(FIG. 1). The Boundless Projected Interactive Virtual Desktop system mayalso include software, such as the Boundless Projected InteractiveVirtual Desktop program 108A, 108B (FIG. 1) that may use the motionsensor 116 (FIG. 1) in the device to pan, scale, and rotate theprojected image as a boundless virtual desktop according to the locationand orientation of the device's projector 112 (FIG. 1). For example, asa user may move the projector device 102A, 102B (FIG. 1) containing themotion sensor 116 (FIG. 1), the portion of the boundless desktop beingprojected may be adjusted.

Next at 204, the location of the device is captured from the motionsensor which includes capturing a change in location, direction, ororientation. As such, the Boundless Projected Interactive VirtualDesktop program 108A, 108B (FIG. 1) can determine the location of thedevice based on the motion sensor that is integrated in the device.However, according to an alternate implementation, the motion sensor maybe implemented as software interpreting images being captured from acamera attached to the projector device as opposed to the motion sensorbeing implemented as a physical piece of hardware integrated in theprojector device. Additionally, an infrared wave point may be used onthe projected surface in accordance with an alternate implementation. Assuch, an infrared wave point may be implemented on the projected surfaceso that the projector device, either through a camera or an infraredsensor, may be able to determine where the projector device is locatedwith respect to the fixed point noted by the infrared wave point.Furthermore, the motion sensor may be selected from a group consistingof an embedded accelerometer; a gyroscope; a positioning system; and anembedded or external electromagnetic or ultrasonic motion sensor.

Then at 206, the projected image is computed (i.e., calculated) based onthe location of the device relative to the projected surface. Aspreviously described with respect to one implementation, the projectionof the image may be calculated by the Boundless Projected InteractiveVirtual Desktop program 108A, 108B (FIG. 1) to take into account anumber of variables, either absolute or relative to a marker, such asthe projected surface. For example, variables may include the locationof the device in three dimensions; the orientation of the device inthree dimensions; and placement of objects on the surface. Using basicthree dimensional geometry, the dimensions of the projected image can becomputed by extending a cone or pyramid from the projector to thesurface. The outer dimensions of the projection are calculated and theprojected image is adjusted to project a corresponding portion of theboundless desktop.

Therefore, based on such variables, the viewing area may be panned andthe projection zoomed by the Boundless Projected Interactive VirtualDesktop program 108A, 108B (FIG. 1). Also, the image of the viewing areamay be stretched and distorted as if the device was flat. Furthermore,an object placed on the surface may be detected and then, the image maybe corrected so that the image may not be magnified by being closer tothe device when the object is placed between the surface and the device.Additionally, to further improve the accuracy, the device 102A, 102B(FIG. 1) may be used in conjunction with table mounted cameras or othersensors (such as an array of infrared light (IR) devices) capable ofpinpointing the location and position of the device in three dimension.

Next at 208, an unbounded projected image is coordinated acrossapplications running in the mobile device based on the location andorientation of the projector. As such, each window or element on thedesktop is positioned with regard to a center reference point. Accordingto one implementation of the present embodiment, the coordination of theunbounded projected image across at least one application running in themobile device may be based on the captured location and an orientationof the projector. Additionally, the strength associated with the imageprojection may be based on a distance of the projector to the projectionsurface.

Then at 210, a bounded view of a portion of the unbounded area isprojected. The projected bounded view may consist of multiple elementsincluding application images representing a virtual desktop. Accordingto at least one implementation, the bounded view may preserve a positionand size of elements in the bounded projection relative to theprojection surface by making adjustments to the calculation of thebounded view. Furthermore, the elements may be selected from a groupincluding documents, images, folders, videos, web browser tabs,application windows, and an application icons.

Next at 212, the bounded view is adjusted based on a criteria. Accordingto one implementation of present embodiment, the criteria that may beused to adjust the bounded view may include a set of rules. The rulesmay be based on elements such as, but not limited to, power consumption;distance to a projection surface; motion; image fidelity; outside lightor ambient light; tilt or angle of the device relative to the surface;and information received from at least one other device. The primaryadjustments to the view keep elements in the projected image at aconstant location, orientation and perspective relative to the projectedsurface in spite of any movement of the projecting device in spite ofmovement of the device. Power consumption and remaining power ismonitored and the image is adjusted corresponding to adjustments inlumen output and projection zoom as described later, keeping theposition of elements relative to the surface while adjusting lamp outputand zoom to decrease or increase the size of the viewing area.

Additionally, the present embodiment may provide a user interface (UI),whereby the user interface (UI) may allow one or more users to use atleast two separate devices to act on the elements independently. Forexample, the user interface (UI) may allow at least two users to act onthe elements independently while each user is using their own separatedevice. Also, the user interface (UI) associated with the boundlessprojected interactive virtual desktop may support gestures selected froma group of gestures. The group may consist of gestures such as drag;pick up; drop; pin; and navigate, etc. For example, as a user moves theprojector device containing the motion sensor, the portion of theboundless desktop being projected may be adjusted. As previouslydescribed, a user may flick the device forward to put an item down(which may create a stack); flick it up to pick up an item from thetabletop; or flick it to the side to shuffle through a stack. As thedevice is moved, the movement may be tracked through motion sensors inthe device. According to at least one implementation, after the boundedview is adjusted based on the criteria, the method may continue in aloop back to step 204, previously described, to capture the location ofthe device from the motion sensor and repeat the process described withrespect to steps 206-212.

It may be appreciated that FIG. 2 provides only an illustration of oneimplementation and does not imply any limitations with regard to howdifferent embodiments may be implemented. Many modifications to thedepicted environments may be made based on design and implementationrequirements. For example, as previously described with respect to analternate implementation, the motion sensor may be implemented assoftware interpreting images being captured from a camera attached tothe projector device as opposed to the motion sensor being implementedas a physical piece of hardware integrated in the projector device.Additionally, an infrared wave point may be used on the projectedsurface in accordance with an alternate implementation. Also, aspreviously described, in addition to a motion sensor 116 (FIG. 1), whichmay aide with movement of the device relative to the surface, the device102A, 102B (FIG. 1) may use techniques from using either active orpassive autofocus detection to detect the relative angle of the plane ofthe projected surface relative to the device. As such, multiple pointscan be sampled for optimal focus, deriving the distance to multiplepoints in the plane of projection and therefore, may allow for thecomputation of the distance to multiple points and recreation of theplane's geometric in the device's memory.

Referring now to FIG. 3, an exemplary illustration 300 of the projecteddesktop image according to at least one implementation of the presentembodiment is depicted. As previously described, a user may use aprojector device 302 that includes a motion sensor 116 (FIG. 1), aprojector 112 (FIG. 1) and software 108A, 108B (FIG. 1) that may use themotion sensor 116 (FIG. 1) in the device 302 to pan, scale, and rotatethe projected image as a bounded view (i.e., a portion) of the overallvirtual desktop 304 created according to the location and orientation ofthe device's 302 projector 112 (FIG. 1). For example, according to atleast one implementation of the present embodiment, a “snapshot” of acomputer screen may be projected and portrayed as an image 304 onto asurface, such as a wall or a desktop. Together, all the areas projectedform a virtual desktop 304. The virtual desktop 304 is capable ofdisplaying a number of “elements” 306-312 which may include documents,images, folders, video, web browser tabs, application windows orapplications. For illustration purposes only, the projected area 304 isshown as a circle, however, it may be portrayed as another shape, suchas a rectangle or a square.

A user using the projector device 302, may interact with the projectedvirtual desktop 304 by using gestures, the on screen display, andbuttons on the device 302. For example, as a user moves the projectordevice 302 containing the motion sensor, the portion of the desktopbeing projected 304 may be adjusted. For example, movement to the rightmay portray a different portion of the projected desktop 304 or suchmovement may include the portrayed portion of the desktop 304 before themovement in addition to portraying the new portion of the projecteddesktop 304.

For illustration purposes, the area outside 314 the virtual desktop 304may depict an area of display which is not visible to a user since it isnot currently being projected. However, this area 314 may be representedinside the projector device 302 and may be revealed by moving theprojector device 302. Additionally, the dot 316 portrayed at the top ofthe projector device 302 represents the location of the projector on thedevice for illustrative purposes.

Referring now to FIG. 4, an exemplary illustration 400 of movement ofthe device adjusting the projected desktop image according to at leastone embodiment is depicted. As previously described, as a user moves theprojector device 302, the portion of the desktop being projected 304 isadjusted. For example, movement of the projector device 302 to the rightmay portray a different portion of the projected desktop 304 or suchmovement may include the portrayed portion of the desktop 304 before themovement in addition to portraying the new portion of the projecteddesktop 304. Therefore, movement of the projector device 302 may resultin the location of the previously displayed “elements” 306-312, such asdocuments, images, folders, video, web browser tabs, application windowsor applications may be adjusted with respect to the portrayed portion ofthe desktop 304 and new “elements” 402-404 may be displayed.Furthermore, as portrayed in FIG. 4, movement of the projector device302 may adjust the position of the portrayed portion of the projecteddesktop relative to the surface 314 upon which the projected desktopimage is being displayed.

Furthermore, the present embodiment may be able to create a virtualsurface 314 using only a small and relatively low power projector whichmay be embedded in the device 302, and may adjust the strength of theprojection based on available battery life without sacrificing theeffective strength. For example, the user may move the device 302 closerto the surface and, although the illuminated area is then smaller, itmay be just as bright with less power consumption in such a situation.As such, battery power will be conserved by the combined adjustment oftotal lighting and lens adjustment, resulting in a preservation of theluminance at the cost of the projection size.

Additionally, according to at least one implementation, when doublingthe distance of a projector 112 (FIG. 1) in a device 302 to the surface314, the surface area 314 of the projected image 304 will quadruple (bydoubling in each dimension) assuming the projector 112 (FIG. 1) uses theprojector's 112 (FIG. 1) full area. Assuming constant lumen output fromthe projector 112 (FIG. 1), the lux (lumens per square meter) of theprojected image will be one fourth for each doubling in distance. Aslong as there is sufficient stored power available as determined by theprojected lifetime of the current battery charge and the users settingsor current use, the present embodiment may increase the lumen outputfrom the light source of the projector 112 (FIG. 1) in response to themovement of the device 302 from the projector surface 314, with theintention of maintaining a constant lux. For instance, ignoring variabletransmittance of light through the projector 112 (FIG. 1) resulting froma different composition of output wavelengths at different dimmingratios, the projector 112 (FIG. 1) may quadruple the lumen output for adoubling in distance of the device 302 to the surface 314.

Furthermore, the device 302 may reduce power consumption whilemaintaining constant lux by reducing the surface area 314 of theprojection, while also maintaining the size of elements 306-312 in theprojected image 304 relative to the surface 314, allowing for an easy toview desktop even in higher light conditions with high batteryconservation. In one embodiment, a liquid crystal on silicon (LCoS)pico-projector is used where the lumens emitted from the light source orlamp are varied in conjunction a lens array used to focus the lightsource to achieve a constant size of projection relative to the surface314. When power conservation is required, the present embodiment mayautomatically reduce the size of the projected surface or focus thelight source by optical elements focusing the total lumens of the lightsource over a smaller projected area 304, while coordinating dimming tothe light source. For instance, with an LED as a light source to power adigital light processing (DLP) or liquid crystal on silicon (LCoS), theinput power to the light source may be dimmed using existing LED dimmingtechniques. The reduction in the area of the projection 304 iscoordinated with a reduction in the lumen output to achieve a constantlux over the projected image. Normally, this alone would result insimply a smaller image being projected onto the surface. For instance, adocument shown on the projected surface 314 would appear half the sizefor each half reduction in the projected image 304 size. The presentembodiment recalculates the image to be projected, adjusting the mappingof the desktop to the projector's image producing elements, effectivelyprojecting a smaller area of the original image directly in relation tothe reduction in projected image size, and preserving the overallrelationship of the projected object to the surface 314. The result forthe end user is simply seeing less of the desktop projected 304 insteadof seeing a smaller or dimmer image as battery conservation isdetermined to be more critical.

Referring now to FIG. 5, an exemplary illustration 500 of coordinationof multiple devices interacting with different subsections of theprojected desktop image according to at least one embodiment isdepicted. According to at least one implementation, the presentembodiment may allow at least one user to use different projectordevices 502, 504 to interact with the elements of the virtual desktopsimultaneously; for example, multiple users, using different projectordevices 502, 504 to interact with the elements of the virtual desktop518 simultaneously. As such, when an element 506-516 is modified, themodifications may be communicated through near field communication,through a server, such as server 114 (FIG. 1), cloud service or throughnetwork communication 110 (FIG. 1) between the devices 502, 504. Assuch, multiple devices 502, 504 may interact either with radio antennaor other positioning sensors to improve their position and orientationrelative to each other. As the projection areas become close andoverlap, cameras on the devices 502, 504 may be used to improve theaccuracy of device location and orientation detection by monitoring theimage projected by the other devices 502, 504 relative to therepresentation of the tabletop in memory. Additionally, visual tokens orelements outside of the human visual spectrum may be embedded in theprojection to assist multiple devices 502, 504 coordinating theirprojections.

The bounds for areas of projection (or device 502, 504 positions,orientations and projector configuration from which areas of projectionmay be calculated) are communicated over near-field communication,including RFID, Bluetooth or communicated through a computer network.Alternatively, each device 502, 504 may independently detect theprojected images 518 of other devices 502, 504 through the use ofcameras, including cameras or detectors outside the field of humanvision. Once an overlap in projected images 518 from multiple devices502, 504 are detected by any device 502, 504, several power control andcoordination mechanisms are used. A single device 502, 504 mayindependently reduce or eliminate the output from its projector lightsource if the device's 502, 504 projected image has significant overlapwith the projection from another device 502, 504. The device 502, 504may coordinate this reduction in output with other devices 502, 504through all the communication methods listed above, sending messages andpossibly allowing for confirmation. Devices 502, 504 may send powerusage and remaining power statistics to allow other devices 502, 504involved with the projection to confirm power reduction strategies, orall data may be sent to a networked device 502, 504 such as a server 114(FIG. 1) which issues commands to the devices 502, 504. In the case of alaser beam steering approach (LBS) projector, the laser may be steeredso as to skip pixels in the overlapping areas of projection, either byturning off when focused on those pixels, or by steering around thoseareas. To maintain a constant frequency or refresh rate to the overallprojection, if the laser is steered around the overlapping areas, thelaser may be turned off for a time corresponding to the amount of timereduced within a cycle by steering around certain pixels or areas of theprojection. In a lamp based solution such as liquid crystal on silicone(LCoS) or digital light processing (DLP), the device may reduce or turnoff the projector's lamp if there is overlap in the projected areas. Thethreshold for the overlap required before these techniques are employedmay be configured and may be dependent on the remaining power and powerusage of the devices involved.

FIG. 6 is a block diagram 600 of internal and external components ofcomputers depicted in FIG. 1 in accordance with an illustrativeembodiment of the present invention. It should be appreciated that FIG.6 provides only an illustration of one implementation and does not implyany limitations with regard to the environments in which differentembodiments may be implemented. Many modifications to the depictedenvironments may be made based on design and implementationrequirements.

Data processing system 800, 900 is representative of any electronicdevice capable of executing machine-readable program instructions. Dataprocessing system 800, 900 may be representative of a smart phone, acomputer system, PDA, or other electronic devices. Examples of computingsystems, environments, and/or configurations that may represented bydata processing system 800, 900 include, but are not limited to,personal computer systems, server computer systems, thin clients, thickclients, hand-held or laptop devices, multiprocessor systems,microprocessor-based systems, network PCs, minicomputer systems, anddistributed cloud computing environments that include any of the abovesystems or devices.

User client computer 102A, 102B (FIG. 1) and network server 114 (FIG. 1)may include respective sets of internal components 800 a,b,c andexternal components 900 a,b,c illustrated in FIG. 6. Each of the sets ofinternal components 800 include one or more processors 820, one or morecomputer-readable RAMs 822 and one or more computer-readable ROMs 824 onone or more buses 826, and one or more operating systems 828 and one ormore computer-readable tangible storage devices 830. The one or moreoperating systems 828 and the Boundless Projected Interactive VirtualDesktop program 108A, 108B (FIG. 1) in client computer 102A, 102B(FIG. 1) and network server 114 (FIG. 1) are stored on one or more ofthe respective computer-readable tangible storage devices 830 forexecution by one or more of the respective processors 820 via one ormore of the respective RAMs 822 (which typically include cache memory).In the embodiment illustrated in FIG. 6, each of the computer-readabletangible storage devices 830 is a magnetic disk storage device of aninternal hard drive. Alternatively, each of the computer-readabletangible storage devices 830 is a semiconductor storage device such asROM 824, EPROM, flash memory or any other computer-readable tangiblestorage device that can store a computer program and digitalinformation.

Each set of internal components 800 a,b,c also includes a R/W drive orinterface 832 to read from and write to one or more portablecomputer-readable tangible storage devices 936 such as a CD-ROM, DVD,memory stick, magnetic tape, magnetic disk, optical disk orsemiconductor storage device. A software program, such as the BoundlessProjected Interactive Virtual Desktop program 108A, 108B (FIG. 1) can bestored on one or more of the respective portable computer-readabletangible storage devices 936, read via the respective R/W drive orinterface 832 and loaded into the respective hard drive 830.

Each set of internal components 800 a,b,c also includes network adaptersor interfaces 836 such as a TCP/IP adapter cards, wireless Wi-Fiinterface cards, or 3G or 4G wireless interface cards or other wired orwireless communication links. The Boundless Projected InteractiveVirtual Desktop program 108A, 108B (FIG. 1) in client computer 102A,102B (FIG. 1) and network server 114 (FIG. 1) can be downloaded toclient computer 102A, 102B (FIG. 1) and network server 114 (FIG. 1) froman external computer via a network (for example, the Internet, a localarea network or other, wide area network) and respective networkadapters or interfaces 836. From the network adapters or interfaces 836,the Boundless Projected Interactive Virtual Desktop program 108A, 108B(FIG. 1) in client computer 102A, 102B (FIG. 1) and network server 114(FIG. 1) is loaded into the respective hard drive 830. The network maycomprise copper wires, optical fibers, wireless transmission, routers,firewalls, switches, gateway computers and/or edge servers.

Each of the sets of external components 900 a,b,c can include a computerdisplay monitor 920, a keyboard 930, and a computer mouse 934. Externalcomponents 900 a,b,c can also include touch screens, virtual keyboards,touch pads, pointing devices, and other human interface devices. Each ofthe sets of internal components 800 a,b,c also includes device drivers840 to interface to computer display monitor 920, keyboard 930 andcomputer mouse 934. The device drivers 840, R/W drive or interface 832and network adapter or interface 836 comprise hardware and software(stored in storage device 830 and/or ROM 824).

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method for creating a boundless projectedinteractive virtual desktop, wherein the boundless projected interactivevirtual desktop comprises an adjustable image of a projected portion ofan area associated with at least one desktop of a computing device, themethod comprising: integrating a projector and a motion sensor into adevice; capturing at least one of a location, a change in location, achange in direction, or a change in orientation associated with thedevice from the integrated motion sensor; computing a projected imagebased on the captured location, the captured change in direction, or thecaptured change in orientation of the device relative to a projectedsurface; coordinating the computed projected image across at least oneapplication running in the device based on the captured location and anorientation of the projector, wherein the projected image is associatedwith the projection surface; projecting a view of a portion of an areaassociated with the coordinated projected image, wherein the projectedview comprises an interactive virtual desktop including a plurality ofelements comprising application images; and adjusting the projected viewof the portion of the area based on a criteria, wherein adjusting theprojected view comprises projecting a smaller portion of the projectedview when a battery conservation is determined to be critical for alight source associated with the projection surface, and wherein theprojected smaller portion of the projected view maintains a size of aplurality of elements associated with an originally projected view. 2.The method of claim 1, wherein the criteria comprises a plurality ofrules based on at least one of a power consumption; a distance to aprojection surface; a motion; an image fidelity; an outside light or anambient light; a tilt or an angle of the device relative to the surface;and information received from at least one other device.
 3. The methodof claim 1, wherein the plurality of elements are selected from a groupconsisting of at least one of a document; an image; a folder; a video; aweb browser tab; an application window; and an application icon.
 4. Themethod of claim 1, further comprising: adjusting a strength associatedwith the projected view based on a distance of the projector to theprojection surface.
 5. The method of claim 1, wherein the motion sensoris selected from a group consisting of at least one of an embeddedaccelerometer; a gyroscope; a positioning system including an autofocussystem; and an embedded or external electromagnetic or ultrasonic motionsensor.
 6. The method of claim 1, wherein the projected view preserves aposition and size of at least one element within the plurality ofelements in the bounded projection relative to the projection surface bymaking adjustments to a calculation associated with the projected view.7. The method of claim 1, further comprising: providing a user interface(UI) associated with the projected interactive virtual desktop, whereinthe user interface (UI) allows at least two users to act on at least oneelement within the plurality of elements independently wherein the firstuser is on a first device and the second user is on a second device andwherein a combination of a projected area associated with the firstdevice and a projected area associated with the second device reveals acontiguous image and a larger image of the portion of the areaassociated with the projected image than the first device or the seconddevice could reveal in isolation of one another.
 8. The method of claim7, wherein the user interface (UI) associated with the projectedinteractive virtual desktop supports at least one gesture selected froma group consisting of at least one of a drag; a pick up; a drop; a pin;and a navigate.
 9. The method of claim 1, wherein the adjusted projectedview is based on a detected change in movement associated with thedevice and the adjusted projected view reveals a previously unrevealedportion of the projected image.